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ABSTRACT 

y 

Adaptive  relaxation  algorithms  use  anti-jamming  schemes  that  require  either  a 

/"■""a  y~  \ 

large  amount  of  computation  or  a  large  amount  of  memory .<vfe  this  paper  vrt  presents 
a  non- adaptive  approach  that  possesses  substantial  computational  and  memory  advan¬ 
tages  over  the  adaptive  schemes.  The  approach  uses  averaging  and  may  be  applied 
whenever  the  relaxation  algorithm’s  point-to-set  maps  satisfy  appropriate  assumptions. 
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I.  INTRODUCTION 

There  exist  many  situations  ([l]-[3],  [5],  [8],  [10],  [12]-[16])  in  which  the  solution 
set  D(P)  (assumed  to  be  in  En  for  simplicity’s  sake)  of  a  problem  P  may  be  expressed 
as  the  intersection  of  the  solution  sets  D  ( Pj )  of  a  finite  number  of  problems  Pj,  j  =  1, 
2,..#,  that  is  D  (. P )  =  D  (Pi)  n  D  (P 2)  n  •  •  •  n  D  (Pp).  In  that  case,  one  may  try  to 
find  a  set  T,  point- to-set  mapsu4y(.),  j  =  1, 2 and  a  surrogate  cost  v  (.)  so  that  the 
following  hypotheses  are  satisfied. 

Hypothesis  1: 

(i)  T  is  a  closed  subset  of  En . 

(ii)  v  (.)  is  from  T  into  E. 

For  j  =  1, 2 

(iii)  Aj(.)  is  from  T  into  all  the  non-empty  subsets  of  T, 

(iv) D(Pj)  =  {z€T\zeAj(z)}. 

Hypothesis  2:  For  j  =  1, 2 if  a  point  z  is  in  7  but  not  in^  (z),  scalars  ey  (z)  >  0, 
Sj(z)  >  0  and  A;(z)  exist  such  that  v  (y ')  <  v  (z ')  -  6j(z),  and  Xj(z)  <  v  (z ')  for  every  y ' 
iaAj (z ')  and  for  every  z '  in  B  (z, ey(z))  n  T. 

Hypothesis  1  ensures  that  the  relaxation  algorithm  given  below  is  well  defined 
and  Hypothesis  2  is  the  usual  monotonicity  assumptions  used  to  obtain  asymptotic  sta¬ 
bility  in  the  large  [4],  [6],  [7],  [11],  [17],  [18]. 


Starting  at  some  point  z  1  in  T,  one  generates  a  sequence  (z,- }  by  using  one  of  the 
maps  Aj(.)  at  iteration  i. 


Algorithm  1:  Let  z  1  in  T  be  given. 
StepO:  Set  /  =  1. 
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Step  1:  Find  a  point x,  inAm  (,)(z,),  l<m  (/)</?. 

Step  2:  If  v  (x,)  >  v  (z,),  let  z,-  + 1  =  z,-,  let  i  =  i  + 1,  and  go  to  Step  1;  else,  let  z,  + 1  = 

X,-,  let  z  =/+l  and  go  to  Step  1. 

If  Algorithm  1  uses  point-to-point  maps  aj(.),  that  is,  ifAj(z)  contains  one 
and  only  one  point  aj(z)  for  every  z  in  T  and  j  =  1, 2 if  those  maps  are  continu¬ 
ous  on  T,  and  if  Hypotheses  1  and  2  are  satisfied,  then  every  cluster  point  of  every 
sequence  generated  by  Algorithm  1  is  in  D  (P),  and  furthermore,  if  D  (P)  contains  at 
most  a  countable  number  of  points,  every  bounded  sequence  generated  by  Algorithm 
1  converges  to  a  point  in  D  (P),  provided  that  {m  (z )}  satisfies  the  following  hypothesis 
proposed  by  Fiorot  and  Huard  [2],  [3,  p.  76,  Hypothesis  H5]: 

Hypothesis  3:  An  integer  r  exists  so  that  to  every  i  =  1, 2,...  and  j  in  the  interval  [l,p] 
correspond  at  least  one  index  k  in  [ij  +r]  such  that  m  ( k )  =  j. 

It  is  not  difficult  to  find  maps  m  (.)  that  satisfy  Hypothesis  3:  a  cyclic 
approach  corresponds  to  letting  m  (1)  =  1,  m  (2)  =  2,...,  m(p)  =  p,m(p  + 1)  = 
p  + 1,...,  etc,  and  Aitken  double  sweep  choice  [4,  p.  158]  corresponds  to  letting  m  (1)  = 
1,  m  (2)  =  2,...,  m  (p)  =  p,  m  (p  + 1)  =  p-1,  m  (p  +2)  =  p- 2,...,  etc.  For  examples  of 
the  approach  used  in  the  context  of  unconstrained  minimization,  see  Luenberger  [4, 
pp.  158-159]. 

Unfortunately,  when  Algorithm  1  uses  point-to-point  maps  that  are  not  con¬ 
tinuous  or  point- to-set  maps,  its  asymptotic  properties  may  not  be  related  to  D(P) 
([3],  [12]).  In  such  cases  one  may  use  one  of  the  two  adaptive  schemes  given  in  [7]  for 
determining  the  quantity  m  ( i )  at  iteration  z.  Both  schemes  have  drawbacks:  either  a 
large  amount  of  computation  or  a  large  amount  of  memory  is  required.  To  alleviate 
those  difficulties,  we  propose  an  algorithm  based  on  averaging  that  uses  a  sequence 
{m  (z)}  that  may  be  chosen  non-adaptively. 


n.  A  NON-AD AFITVE  AVERAGING  SCHEME 


The  asymptotic  properties  of  the  averaging  algorithm  are  related  to  D  (P) 
when,  in  addition  to  Hypothesis  1,  the  following  hypothesis  is  satisfied. 

Hypothesis  4: 

(i)  The  set  T  is  convex. 

(ii)  The  map  v  (.)  is  upper-semi  continuous  on  T  with  respect  to  T,  that  is,  to  every 
point  z  in  T  and  6  >  0  corresponds  an  e  >  0  such  that  v  (z  ')<  v  (z)  +  6  for  every  z '  in 
B  ( 2 ,  e)  n  T. 

For  every  index  j  in  {1, 2 

(iii)  if  a  point  z  is  in  T  but  not  inAj(z),  scalars  ey(z)  >  0, 6j(z)  >  0  and  Xj(z)  exist  such 
that  v(z'+p(y'-z '))  <v(z')-  pSj(z),  and  \j(z)  <  v  (z ')  for  every  p  in  the  interval 
[0,1],  for  every  y '  in  Aj(z ')  and  for  every  z '  in£  (z,e;(z))  n  T, 

(iv)  if  a  point  z  is  inAj(z),  then  to  every  scalar  6  >  0  corresponds  a  scalar  e  >  0  such 
that  v  (z)  -  S  <  v  (y ')  for  every  y '  in  Aj(z ')  and  for  every  z'  mB(z,e)nT. 

The  averaging  algorithm  uses  an  initial  guess  of  a  solution  z  i  in  T  and  a 
sequence  {m  (i)}  that  takes  its  values  in  the  set  {1, 2,...,p}. 

Algorithm  2:  Given z\  in  T  and  { m  (/)}  in  {1, 2 ,...,/?} 


Step  0:  Set  i  =  1. 

Step  1:  Find  a  pointy,  in^4m  (,)(z,). 

Step  2:  If  v  (y, )  >  v  (z, ),  let  z,  + 1  =  z,-,  let  i  ■  r  + 1  and  go  to  Step  1;  else  go  to  Step  3. 


Step  3: 


Let  m  =  min 


1, 


v(z,)-v(y,)' 
lb«*«ll  , 


Step  4:  Let  z,  +i  =  z,  +p,(y,-z,),  let  i  -  i  + 1  and  go  to  Step  1. 

Lemma  1:  If  Hypotheses  1  and  4  are  satisfied,  then  Algorithm  2  is  well  defined,  and 
whenever  z,  + 1  and  z,  are  two  consecutive  points  of  a  sequence  generated  by  the 


-5- 


algorithm  such  that z,  +i  ^z„  thenv(zl+1)  <  v(z,),and  ||zl+rz,||  <v(z,)-v(yl). 

Proof:  Parts  (ii)  and  (iii)  of  Hypothesis  1,  Part  (i)  of  Hypothesis  4,  and  the  fact  that  if 
v  (y,)  <  v  (z, ),  the  quantity  m  is  in  [0,1]  immediately  imply  that  the  algorithm  is  well 
defined  If  z,-  + 1  and  z,  are  two  consecutive  points  of  a  sequence  generated  by  Algo¬ 
rithm  2,  and  if  z,  +  i  ^z,-,  then  it  is  clear  that  v  (y,)  <  v  (zf),  m  ?0,zi+l  =  z,  +/j,(yt-zt  ) 
and  from  (iii)  of  Hypothesis  4,  v  (z,  +1)  <  v  (z,  ).  If/i,  <  1,  then  ||z1+1-zf 1|  = 
v (zi)~v O'/)*  if  Mf  =  1,  thenz,  +1  =  y,-,v(z,)-v(yl)>  ||y,-z,||,andthus,ifzl+1  +zt, 

v (Zi)-v 0.)  >  Ibr*.  II  =  Ik  +i-z«  II-  □ 

We  now  analyze  the  asymptotic  properties  of  Algorithm  2. 

Lemma  2:  Suppose  that  Hypotheses  1  ami  4  are  satisfied  and  let  {z,  }  be  a  sequence 
generated  by  Algorithm  2.  If  an  infinite  subset  K  of  the  integers  exists  so  that  (i)  the 
subsequence  {z,  }*■  converges  to  some  point  z*,  (ii)  the  subsequence  (y,  }jf  is  bounded 
and  (iii)  m  (i)  =  j  for  every  index  i  i nK,  thenz*  belongs  to  Aj(z»),  the  subsequence 
{ l|z,  + 1  -  z,  ||  }jf  converges  to  0,  and  the  subsequence  (z,  +  x }x  converges  to  z.. 

Proof:  The  set  T  is  closed  the  sequence  {z,}  is  in  T,  and  thus  z.  is  in  7.  Assume  that 
z*  does  not  belong  to  Aj(z»).  Part  (iii)  of  Hypothesis  4  and  the  fact  that  m  ( i )  =  j  for 
every  index  i  in  K  imply  that  an  index  k  and  a  scalar  $;(z*)  >  0  exist  such  that 

v  ( z,  +rty,-z,))  <  v  (z{)  -  ti 6j(z»)  (3) 

for  every  n  in  [0,1]  and  for  every  i  >  k,  i  in  K. 

Using  Eq.  (3)  with  n  =  1  yields 

v(y,-)<v(z,)-$;-(z.).  (4) 

Thus,  for  every  i  >k,  i  inK,  v(y,  )  <  v  (z,  )  and  m  is  computed  in  Step  3  of  Algorithm  2. 
The  subsequences  {z,  }*  and  {y,  }^  are  bounded  and  thus  a  scalar  A  >  0  exists  so  that 
|| yi-Zi  ||  <  A  for  all  /  in  K.  Using  Eq.  (4)  we  may  immediately  conclude  that  for  every  i 


It  follows  that  for  every  i  >  k,  i  in  K 


V  (z,  +Vk(yrZi))  <  V  ( Z, )  -  UmiaSjiz.) 
and 

*'fe+i)<v(zl)-/imintf/(z.)i  (5) 

From  Lemma  1  we  know  that  v  (z,  + i )  <  v  (z, )  for  every  index  i  and  thus  Eq.  (5) 
implies  that  the  sequence  {v  (z, )}  is  unbounded  from  below.  This  is  not  possible  in 
view  of  Hypothesis  4,  pan  (iii),  and  we  must  conclude  that  z.  belongs  toAj(z»). 
Suppose  now  that  { |jz,  +  j  -z,  j|  )x  does  not  converge  to  0,  that  is,  suppose  that  an 
infinite  subset  M  of  K  and  a  scalar  A  >  0  exist  so  that  ||zl+i-z,  ||  >  A  for  every  i  in  M. 
Lemma  1  implies  that 

v(Zi)-v(yj)>  ||z,-+i‘Z,  ||, 
and 


v(y,)<v(z,)-A.  (6) 

The  point  z*  is  inv4;(z*),  and  pan  (iv)  of  Hypothesis  4  implies  the  existence  of  a  scalar 

e  >  0  such  that  v  (z.)  -  —  <  v  (y )  for  every  y  inAj(z)  and  for  every  z  in  B  (z»,e)  n  T. 

The  subsequence  {z,}^  converges  toz.,  the  pointy,  is  in/!;(z,)  for  every  index  i  in  M, 
and  therefore  an  integer  m  exists  so  that  for  every  i  >  m,  /  in  M,  z,  is  in  B  (z.,e)  n  T 
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v(z.)- y<v(y,). 


(7) 


Using  Eqs.  (6)  and  (7),  we  obtain 


v(z«)  +  y<v(z,-) 


(8) 


for  every  i  >m,i  in  A/.  Eq.  (8)  contradicts  part  (ii)  of  Hypothesis  4  and  we  must  con¬ 
clude  that  { ||z,-  +  1-z;  ||  }*  converges  to  0.  The  subsequence  {z,}*  converges  toz.  and 
therefore  the  subsequence  {z,  +1}*  converges  toz*  also.  □ 

Lemma  3:  Suppose  that  Hypotheses  1  and  4  are  satisfied  and  let  {z,}  be  a  sequence 
generated  by  Algorithm  2.  If  an  infinite  subset  K  of  the  integers  exists  so  that  the 
subsequence  {z,  }*  converges  to  some  point  z.  and  the  subsequence  {y,  }*  is  bounded, 
then  the  subsequence  (z,  + 1 }*  also  converges  toz*. 

Proof:  Let  the  sets  K  (j),  j  =  1, 2,...,  p,  J  and  L  be  defined  as  follows: 

K(j)  =  {ieK\m(i)=j) 

•7-01  ^  0)  contains  infinitely  many  indices  } 

L  =  u  K(j) 

/€/ 

It  is  clear  that  {z,}*(/)  converges  toz.  for  every  j  in/.  Lemma  2  implies  that 
{z,  + 1  }x(j)  converges  to  z*  for  every  j  in/,  and  thus  {z,  + !  }i  converges  to  z*.  The 
definition  of  L  implies  that  an  integer  k  exists  so  that 

{*  G  K  |  i  >k)  =  {i  €  L  |  i>k) 
and  thus  the  subsequence  {zi  +1}k  converges  toz*.  □ 

The  sequence  {m  (/)}  cannot  be  arbitrary  if  the  asymptotic  properties  of 
Algorithm  2  are  to  be  related  to  D  (P).  Known  results  ([2],  [3])  indicate  that  {m  (/)} 
should  not  only  take  on  each  value  j,  j  *  1, 2,...,  p  for  infinitely  many  indices  i,  but  that 


s’  /.v  ■  i* *,  * i* *  r*  i* *  I* k~r  p*  iVr  v*  is 

'i  •{*  V.-?V  V  -i*,’!* 


-8- 


the  "density”  of  occurrence  of  these  values  must  not  become  vanishingly  small,  that  is, 
the  map  m  (.)  must  satisfy  Hypothesis  3. 

Theorem  1:  Assume  that  Hypotheses  1, 3  and  4  are  satisfied,  and  let  {z,}  be  a 
sequence  generated  by  Algorithm  2.  If  the  sequence  {y,}  is  bounded,  then  {z,}  is 
bounded,  every  cluster  point  of  {z,}  is  in  D  (P),  and  {z,}  is  asymptotically  regular,  that 
is,  the  sequence  { ||z;  +1  -  z,- 1| }  converges  to  0. 

Proof:  The  definition  of  Algorithm  2  implies  thatz,-  +1  is  in  the  convex  closure  of  the 
set  of  points  z  i ,  y  i ,  y  2>-,  y« .  and  thus  the  boundedness  of  {y, }  implies  the  bounded¬ 
ness  of  {z, }.  Let  z.  be  a  cluster  point  of  a  sequence  {z,}  generated  by  Algorithm  2. 

An  infinite  subset  K  of  the  integers  exists  so  that  the  subsequence  {z,  }*  converges  to 
z*.  Hypothesis  3  implies  that  an  integer  r  exists  such  that  {1, 2 =  {m  (i), 
m  (i  + 1),...,  m  (i  +r)}  for  every  positive  integer  i.  Lemma  3  implies  that  the  subse¬ 
quences  {Zi  +i}*,  {z,  +2>j {z,  +r}jf  converge  toz*.  Let  the  sets  L  an dK(J),  j  =  1, 
2,...,p  be  defined  as  follows: 

L  =  {i  |  |  } U  •••  U{i  |  i-r€K}, 

K(j)  =  {ieL\m(i)=j}. 

The  definition  of  L  implies  that  if  /  is  in K,  then  i,  i  + 1,  i  +2,...,  and  i  +r  are  in  L.  Thus 
to  every  j  in  {1, 2 and  i  in  K  correspond  an  index  i  (j)  in  L  such  that  i  (j)  >  i , 
i(J)  -  i  <  r,  and  m  (i  (/))  =  j.  It  follows  that  K (j)  contains  infinitely  many  indices  for 
every  j  in  {1, 2 and  using  Lemma  2,  we  may  then  conclude  that  z*  is  in^4;  (z.) 
for  every  j  in  {1, 2 and  thus  in  D  (P).  The  proof  that  {z,  }  is  asymptotically  regu¬ 
lar  is  similar  to  the  proof  that  {||z)  +1  -  z,  ||  }*  converges  to  0  in  Lemma  2  and  has  been 
omitted.  □ 

The  cluster  point  set  of  an  asymptotically  regular  and  bounded  sequence  is 
not  arbitrary:  it  contains  either  one  point  or  an  uncountable  number  of  points.  This 
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result,  shown  by  Ostrowski  [9,  p.  173]  may  be  used  to  show  that  sequences  generated 
by  Algorithm  2  converge. 

Corollary  1:  Assume  that  Hypotheses  1, 3  and  4  are  satisfied.  If  D  (P)  contains  at 
most  a  countable  number  of  points  and  if  T  is  bounded,  then  every  sequence  {z,}  gen¬ 
erated  by  Algorithm  2  converges  to  a  point  in  D  (P). 

Note  that  it  is  possible  to  parametrize  Algorithm  2  by  a  scalar  a  >  0.  Let  m 
in  Step  3  of  Algorithm  2  be  defined  by 


.  f,  o(v  (z,)-v  (y,))' 

"■"THEiii  T 

Theorem  1  and  Corollary  1  hold  for  every  a  >  0,  and  therefore  it  is  possible  to  control 
when  the  anti-jamming  feature  of  Algorithm  2  takes  effect  through  the  choice  of  a. 

in.  CONCLUSION 

Relaxation  algorithms  may  be  used  whenever  the  problem  to  be  solved  exhi¬ 
bit  the  appropriate  structure,  but  anti-jamming  schemes  may  have  to  be  present  to 
insure  the  desired  convergence  properties.  To  illustrate  that  point,  we  consider  now 
the  coordinate  descent  method  for  unconstrained  minimization. 

Let  T  =  En,  and  given  a  continuously  differentiable  map  v  (.)  from  T  into  E, 
and  n  linearly  independent  vectors  ej,  j  -  1, 2,...,  n  in£",  let 


and  for  j  =  1, 2,...,  n,  let 


D(P)  =  {z  |  Vv(z)  =  0} 


D(Pj)  =  {z  |  <Vv(z),d;>  =0}. 


Thus,  problem  P  consists  in  finding  a  point  in  D  (P)  and  it  is  clear  that  a  point  z  is  a 
solution  of  P  if  and  only  if  it  is  a  common  solution  of  the  problems  Pj,  j  =  1, 2,...,  n, 
where  each  problem  Pj  has  a  solution  set  D  (Pj).  For  every  j  =  1, 2,...,  n,  let  v4y(.)  be 
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the  map  from  T  into  all  the  subsets  of  T  defined  by 

Aj(z)  =  (y  =  z+Ady  |  <Vv(y),dj>  =  0}. 

Givenz  in  T  and  j  in  {1,2,... /i},  it  is  possible  to  obtain  points  in  Aj(z)  by  minimizing 
v  (.)  over  the  line  passing  through z  with  direction  dj.  Coordinate  descent  methods 
for  minimizing  v  (.)  consists  in  using  Algorithm  1  with  p  =  n  and  a  sequencing  map 
m  (.)  that  satisfies  Hypothesis  3.  Such  methods  are  useful  when  the  structure  of  v  (.) 
is  such  that  its  minimization  along  the  privileged  directions  dj  is  easy.  Although  it  is 
sometimes  believed  that  such  methods  produce  the  desired  results,  the  examples  given 
in  [3]  and  [12]  show  that  jamming  may  occur,  that  is,  the  sequences  (Vv(z,)}  that 
correspond  to  the  sequences  {z,  }  generated  by  Algorithm  1  may  be  bounded  away 
from  0.  To  prevent  jamming,  one  may  use  either  one  of  the  two  adaptive  schemes 
given  in  [7]  or  the  averaging  scheme  presented  in  this  paper.  The  non-adaptive 
schemes  have  a  costly  overhead  that  results  from  having  to  determine  m  (i)  at  every 
iteration  i.  The  scheme  (Algorithm  2)  presented  in  this  paper  is  non-adaptive:  {m  (i )} 
is  given  and  thus  the  overhead  needed  to  implement  anti-jamming  consists  only  in  the 
determination  of  the  appropriate  step  length.  The  efficiency  gained  by  using  averag¬ 
ing  is  obtained  at  a  price:  the  assumptions  that  insure  the  convergence  of  Algorithm  2 
are  stronger  than  the  assumptions  that  insure  the  convergence  of  the  adaptive  algo¬ 
rithms  given  in  [7].  Fortunately,  Hypothesis  4  is  satisfied  in  most  cases  and  the 
averaging  scheme  is  therefore  applicable.  For  example,  the  reader  may  verify  that  the 
averaging  scheme  can  be  used  to  prevent  jamming  when  the  coordinate  descent 
method  is  used  to  minimize  the  three  functions  proposed  by  Powell  in  [12]. 
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